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KXIT  PROBABILITIES  AND  OPTIMAL  STOCHASTIC  CONTROL 


by 


Wendell  II.  Fleming 

Abstract ; This  paper  is  concerned  with  Markov  diffusion  processes 

which  obey  stochastic  differential  equations  depending  on  a small 
£ 

parameter  The  parameter  enters  as  a coefficient  in  the  noise 

term  of  the  stochastic  differential  equation.  The  Ventcel-Freidl in 
estimates  give  asymptotic  formulas  (as  t >•  O)  for  such  quantities 
as  the  p)^obafc>ility  of  exit  from  a region  D through  a given  portion 

N of  the  boundary  Dd,  the  mean  exit  time,  and  the  probability  of 

A 

exit  by  a given  time  T.  A new  method  to  obtain  such  estimates  is 
given,  using  ideas  from  stochastic  control  tlieory. 
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EXIT  PROBABILITIES  AND  OPTIMAL  STOCHASTIC  CONTROL 

Wendell  H.  Fleming 

Accompanying  Statement 

This  paper  is  concerned  with  Markov  diffusion  processes 
which  obey  stochastic  differential  equations  depending  on  a 
small  parameter  £.  The  parameter  enters  as  a coefficient  in 
the  noise  term  of  the  stochastic  differential  equation.  The 
Ventcel-Freidlin  estimates  give  asymptotic  formulas  (as  e ->•  0) 
for  such  quantities  as  the  probability  of  exit  from  a region  D 
through  a given  portion  N of  the  boundary  9D,  the  mean  exit  time, 
and  the  probability  of  exit  by  a given  time  T.  A new  method  to 
obtain  such  estimates  is  given,  using  ideas  from  stochastic 
control  theory. 

The  method  involves  a logarithmic  transformation  on 

positive  solutions  g (x)  of  the  elliptic  partial  differential 

equation  = 0,  where  is  the  generator  of  the  diffusion 

£.  £ 

process.  In  fact,  J = -e  log  g satisfies  a nonlinear  elliptic 
partial  differential  equation,  which  is  just  the  dynamic  programming 
equation  associated  with  the  stochastic  control  problem.  The 
boundary  data  ^ (x)  for  J^(x)  are  chosen  according  to  a "penalty 
function"  method;  for  example,  if  it  is  desired  that  exit  occur 
through  N C (9D,  then  we  take  4>(x)  = 0 on  N and  ^(x)  large 
on  compact  subsets  of  the  complement  of  N.  When  e = o the 
dynamic  programming  equation  reduces  to  the  Hamilton-Jacobi  equation 
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Accompanying  Statement  (Continued) 


Page  2 


for  the  calculus  of  variations  problem  which  enters  in  the  Ventcel- 
Freidlin  estimates.  Thus,  the  stochastic  control  method  provides 
not  only  a different  way  to  prove  sucli  estimates,  but  also  a differ- 
ent intuition  to  explain  why  one  should  expect  them  to  be  true. 
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f EXIT  PROBABILITIES  AND  OPTIMAL  STOCHASTIC  CONTROL 

1 

I 

( 

S Wendell  H.  Fleming 

{ 

i 

1.  Introduction.  Consider  a Markov  diffusion  process 
C on  n-dimensional  R*^  which  obeys  the  stochastic  differential 
equations 

(1.1)  dC*^  = b[5^(t)]dt  + /e  a[4^(t)]dw,  t > 0, 

£ 

with  initial  data  C (0)  = x.  Here  w is  an  n-dimensional 

brownian  motion  and  e a positive  parameter, 
t c 

Let  tp  denote  the  exit  time  of  ? (t)  from  an  open  bounded 
set  D C R*^.  The  Ventcel-Freidlin  estimates  give  asymptotic 
formulas  (as  c-  ->■  0)  for  such  quantities  as  the  probability 

C C G 

P (f^  ( I ) E N)  , where  N C 3D,  the  mean  exit  time  E x , and  the 

XU  XU 

probability  P (x^  < T)  of  exit  by  a fixed  time  T.  See  [16], 
[17],  [7,  II,  Chap.  14].  In  this  paper  we  give  a new  method  to 

obtain  such  estimates,  using  ideas  from  optimal  stochastic  control 
theory.  However,  the  paper  is  self-contained  as  far  as  knowledge 
of  stochastic  control  is  concerned. 

There  are  two  kinds  of  Ventcel-Freidlin  estimates.  The  first 
kind  give  lower  estimates  for  the  probability  that  ?^(t)  remains 
in  a given  open  set  of  curves  for  0 < t < T;  see  [7,  II,  p.  332]. 
These  lower  estimates  follow  rather  easily  from  the  Girsanov 
transformation  formula.  At  the  end  of  the  paper  we  outline  a 
somewhat  different  derivation  of  a lower  estimate. 
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2 

The  second  kind  of  Vontcel-Freidlin  results  give  upper 

estimates  for  the  probability  that  s^(t)  remains  in  a given 

closed  set  of  curves  [7,  IT,  p.  334-345].  The  proofs  are  more 

technical.  Our  main  results  provide  estimates  of  the  second  kind, 

which  suffice  to  study  the  problem  of  exit. 

We  begin  in  §2  by  making  a logarithmic  transformation  on 

positive  solutions  g (x)  of  the  elliptic  partial  differential  j 

equation  ^ q - 0 in  some  region  A,  where  ^ is  the  generator 

e c.  e 

of  the  diffusion  process  K . In  fact,  let  J = -c  log  g . Then 
£ 

J (x)  satisfies  the  nonlinear  partial  differential  equation  (2.5), 

e B 

with  given  boundary  data  J = $ on  3D.  Theorem  2.1  states 

e 

that  J (x)  is  the  minimum  for  a certain  stochastic  control 
problem,  in  which  the  drift  coefficient  b in  (1.1)  is  replaced 
by  a control  process  v(t).  A similar  logarithmic  transformation 
was  used  by  E.  Hopf  [10]  to  solve  Burger's  equation.  Recently, 

(and  independently  of  our  work)  Holland  [8],  [9]  used  a 2 

j 

logarithmic  transformation  of  solutions  to  second  order  linear 
elliptic  equations.  He  obtained  a stochastic  control  representation 
of  the  dominant  eigenvalue  for  Schrodinger ' s equation,  and  another 
proof  of  the  Donsker-Varadhan  formula  for  the  dominant  eigenvalue 
in  case  of  natural  boundary  conditions. 

The  Ventcel-Freidlin  estimates  involve  minimizing  the 

i 

following  functional  I('[>,0)  for  various  choices  of  R'^-valued 

1 

functions  (t)  and  0 > o.  Let 

(1.2)  !(<(>, 0)  = L[<(>  (t)  ,<[>  (t)  ]dt, 

Jo 

I 

where  for  x,  v e R*^ 

I* 

!l 
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(1.3)  L(x,v)  = (b(x)  - V)'a(x)  ^(b(x)  - v) , 

a(x)  = a(x)o'(x). 

In  particular,  lot  A be  open,  bounded,  with  x = '!>  (0)  in  A; 
and  let  0 be  the  exit  time  of  <}>  (t)  from  A.  For  N C 3A 
let  I^(x,N)  denote  the  infimum  of  I('1’,0)  subject  to  the 

I 

additional  condition  '!’('••)  ^ N.  Theorem  5.1  implies  that 

-I.(x,N)  = lim  f loy  t N)  , 

t'.vO 

under  some  rather  stringent  hypotheses  on  b and  A.  As  in 
Icnown  [16],  [17],  [7,  II,  pp.  386-3R7]  one  can  then  apply 
Theorem  5.1  to  get  results  on  the  exit  place  and  exit  time  of 

r C 

(t)  from  a region  D,  under  certain  assumptions  on  the  behavior 
as  t of  solutions  of  the  unperturbed  system  C = b[C  (t)  ] . 

We  include  one  such  result  in  §6. 

In  S3  we  consider  instead  of  (1.2)  the  corresponding  func- 
tional obtained  by  adding  a term  'J’[‘]’(^M]  on  the  right 

side  of  (1.2).  If  'l(y)  = 0 for  y c N and  > 0,  we  may 
regard  'I'[4'(0)]  as  a penalty  imposed  if  ('M  / N at  the  exit 

time  . Later  (in  S5)  wo  ta)co  a seauence  of  penalty  functions 

sucli  that  ‘ ■*'  M * for  y / N. 

In  S7  we  consider  the  nonautonomous  form  of  (1.1)  on  a finite 


time  interval  s < t < T.  There  is  a considerably  simpler  proof 
of  the  main  result  (Theorem  7.1);  and  no  assumptions  on  the  large- 
time behavior  of  the  unperturbed  system  are  needed. 
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2.  The  logarithnic  transformation.  With  the  stocliastic 


differential  equation  (1.1)  is  associated  the  differential  generator 


, namely. 


Here  a = (a.,)  = oo',  and  g,  is  the  gradient.  We  assume: 

J K X 

(A^)  b(x),o(x),  and  the  inverse  o ^(x)  are  l^ounded, 

Lipschitz  functions  on  R*^. 


Since  0 is  bounded,  1/  is  a uniformly  elliptic  operator  for 
each  E > 0. 

n 2 

Let  A C R be  an  open,  hounded  set,  with  C boundary  SA 

2 2 

(i.e.,  3A  is  a manifold  of  class  C .)  Let  be  of  class  C , 

and  41  > 0.  Consider  the  boundary  value  problem 


(2.1) 


= 0 in  A 


(2.2) 


g'^(x)  = exp[-  on  9A 


t ? 1 

There  is  a unique  solution  g , of  class  C'(A)  (A  c (^) , 
= A U 3A.  See  [11,  Chap.  Ill,  §12,  15].  Moreover, 


(2.3)  g (x)  = Kj,exp| J, 


fc  A r ^ 

where  ^ ^ is  the  exit  time  from  A of  s (t) . 

We  ma)ce  the  following  logarithmic  transformation.  Let 
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(2.4: 


c c 

J (x)  = -e  log  g (x) . 


By  elementary  calculus,  J satisfies  in  A the  nonlinear  elliptic 
equation 


(2.5) 


3,k=l  -*  3 k I 


where  for  each  x and  row  vector  p 


(2.6) 


H(x,p)  = - ^ pa(x)p'  + p*b(x) 


Then  H(x,*)  is  dual  to  L(x,*)/  where  L was  defined  by  (1.3),  in 
the  sense  of  duality  for  concave  and  convex  functions.  In 
particular,  from  (1.3)  and  (2.6) 


(2.7) 


H(x,p)  = mintL(x,v)  + P'v] 

V 


Equation  (2.5)  is  the  dynamic  programming  equation  for  the 
following  optimal  stochastic  control  problem.  Let  n(t)  denote 
the  state  of  a system  being  controlled  for  t > 0,  and  v(t)  the 
ccffitrol  used  at  time  t (n(t)  e r”,  v(t)  £ for  each  t)  . These 

processes  are  defined  on  some  probability  space  (Q,  ^,P)  and  are 
nonanticipative  with  respect  to  some  increasing  family  { 3^^} 
of  a-algebras  , ^ C We  assume  that  the  control  v(t) 

is  bounded.  The  state  n satisfies  the  stochastic  differential 
equation 


(2.8) 


dn  = v(t)dt  + /e  o[ri(t)]dw 


6 


with  n(0)  = X,  wherR  x * A and  in  soino  brownian  motion 

adapted  to  { . 

Lot  ^ denote  the  exit  time  of  n(t)  from  A,  and  let 

(2.9)  /^(x,v)  = r,jf  L[n(t)  ,v(t)  ]dt  + .Mn(0)i|. 

'Jo 

The  stochastic  control  problem  is  to  minimize  ' (x , v)  given  the 
initial  state  x. 

e 

From  the  fact  that  J (x)  is  a solution  in  class 
2 1 

C (A)  n C (A")  of  the  dynamic  programming  equation  (2.5),  with 
J = 't  on  3A , one  can  show  that  J^(x)  is  the  minimum  of 

e 

^ (x,v)  and  can  characterize  an  optimal  control  process  v (t) . 
This  result  is  called  a verification  Theorem  [6,  p.  164],  and 
is  stated  below  as  Theorem  2.1.  For  completeness  we  include  the 
easy  proof,-  as  follows. 

From  (2.5)  and  (2.8) 

e ? e c 

(2.10)  5 I V + + L[n(t),v(t)]  > 0, 

j,k=l  ^ k 

e t 

where  a..  ,J  ,J  are  evaluated  at  n(t).  VJe  apply  the  Ito 

I K X • X . X 

^ e 

stochastic  differential  rule  to  J [n(t)]  and  take  E dt: 

J 0 

,0 

(2.11)  j‘'(x)  < E L[n(t)  ,v(t)  ]dt  + EJ^(n(0)]. 

•J  0 

. c 

Since  J =4’  on  DA, 


(2.12) 


G G 

J (x)  < ^ {x,v)  . 


!P 
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If  V is  a function  from  A into  wo  say  that  the 

control  process  v(t)  is  obtained  from  the  feedback  control  law  V, 
for  initial  data  n(0)  = x,  if 


(2.13) 


v(t)  = v[n(t) ] , 0 < t < 0 . 


For  t > ® we  set 
admit  any  feedback 
on  A , 


v(t)  = 0;  this  does  not  affect  /^(x;v).  VJe 
V which  is  bounded  and  locally  Lipschitz 


In  particular,  let 


(2.14)  V^(x)  = H(x,J^(x))  = b(x)  - a(x)J^(x)'. 

p X X 

e e 

Let  V (t) , n (t)  be  the  corresponding  control  process  and  solution 
of  (2.8),  for  given  initial  data  n^(0)  = x.  Equality  holds  in 

G G 

(2.10)  when  v = v = n . Then  equality  also  holds  in  (2.11) 

G G G 

when  0=0,  with  9 the  exit  time  of  n from  A.  Therefore, 


(2.15) 


J*'  (x)  = (x;v^)  . 


From  (2.12)  and  (2.15)  we  have: 

G G G 

Theorem  2.1.  J (x)  is  tlie  minimum  of  ^ (x,v)  , and  V is 
an  optimal  feedback  control  law. 

3.  A deterministic  minimum  problem.  When  e = 0 we  consider 
the  following  minimum  problem.  Let  C^[0,»)  denote  the  space  of 
continuous,  R^-valued  functions  on  [0,«>)  , and  the  space  of 


}) 


<T  1 1 r>urh  Minf:  4'  is  nt)so  ] iit:c'l.v  nno»is  uml 

fT 


[ < ■"  F(ir  I'.u'h  T > 0.  Tor  ^ niul  0 > 0 let 

J 0 

0 

= f I.| 

J n 


(1.1) 


.(9{t)  (t)]  (It  ^ -Ml.  (u)  I . 


I/'t  A I'O  OIKMI  hdvuulod,  with  C*"  houtulnry  i'A ; and  lot  be  of 

2 

class  C ,i  > 0.  nivon  x ' A lot 


(3.2) 


d(x)  = inf  /(')’,*'), 
(|>  , u 


whore  the  infimuin  is  talion  amontj  all  t , " siu'li  I h.it  t (0)  = x, 

']’ ( t ) ' lor  13  i 1”  1 , 'l’(*')  r 3A . Unch'r  condition  (Mj)  below 

the  minimum  is  attained  in  (1.2),  but  wi'  sliall  not  use  this  fact. 
Note  tliat  if  wo  set  ' = 0,  ii(t)  = 9(t),  v(l)  = )’(t)  in  (2.8), 

then  (2.9)  reduces  to  (1.1).  f^oreovor,  when  ' = 0 the  dynamic  pro- 
(pramminq  equation  (2.9)  reduces  to  0 = II(x,,I^).  Tliis  is  tlie 
llami  1 ton-.lacohi  equation  associated  witli  (1.2).  One  might 
expect  thori'fore  tliat  J*'(x)  tends  to 

iT(x)  as  ' ‘ 0.  We  shall  jirove  in  f' 4 a partial  result  of  tliis  kind. 

See  hemma  4.2  and  tlie  Note,  end  of  S4. 

Tn  li  ' s I-*)  wf'  make’  the  following  assumption  about  the  function 
1)  and  tlu’  rc*<iion  A. 

(hj^)  If  i'  ' and  t ( t ) t.  A for  all  t ^ 0,  tlien 

h ft  ( f ) f >1’  ( t)  I dt  = +■"  . 


IiOt  A^(t;x)  denote  the  solution  of 


(1.1) 


= b [(.  (t;x)  I , 


t > 0 
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with  ^^(0;x)  = X.  Note  that  (3.3)  is  the  unperturbed  version  of 
(1.1),  with  e = 0.  Let  us  show  that  (R^)  ic  implied  by  the 
following  property  (02)*  for  u > 0,  let  = {y:  dist(y,A)  > u), 

(B2)  There  exist  > 0,  a > 0 with  the  following  property: 

for  everv  x i:  A , there  exists  t e [0,Tj^]  such  that 
^°(t;x)  e A^^.  , 

Lemma  3.1.  (B2)  implies  (B^^)  . 

Proof . Suppose  tliat  e and  (|)(t)  e A for  all  t > 0. 

Let  be  as  in  (D2)  ; and  for  j = 0,1,2,...  lot 


?j(t)  = f.^(t;(|)(jT^))  , -j'.(t)  = 4>(t+jT^) 


Let  M be  a Lipschitz  constant  for  b,  and  let  | | | |^  denote  the 

sup  norm  on  (0,t].  Then,  for  0 < t < T^^, 

<!).(t)  - i^{t)  = f^[‘f  .-b((|>  .)  Ids  + f^[b(<}>.)  - b(??)]ds, 

J J } Q 3 J )q  3 3 


“ t ^ t 

j - ^jllt  - " b(<l>j)|^ds  + M 


Moreover,  by  (1.3',  |v-b(y)|  < CL(y,v)  for  some  C.  By 

Gronwall's  inequality 


1 

with  C,  = (CT,)^exp  MT.  . However,  ^ < | | ^ I | „ by  (B_)  , 
i i 1 ~ J 3 Tj^  Z 


with  X ro>)lacocl  by  since  I . (t)  c.  A for  0 t < 


Thus 


2 -2 


( j + l)T. 


L('t'  ,'P)dt,  j = 0,1,2,...  . 


Hence,  L('J>,'P)dt  = +''^'  , which  proves  Leinma  3.1. 

J 0 

Lemma  3.1  has  the  following  simple  interpretation.  Condition 
(n^)  says  that  a particle  following  the  flow  (3.3)  is  swept  out 
of  ^ in  bounded  time.  If  L('(>,t)  is  interpreted  as  a rate 
at  which  energy  is  expended  to  resist  the  flov^?,  then  infinite 
energy  is  required  to  remain  in  A indefinitely. 


4.  A semicontinuity  result.  In  this  section  we  consider 
lower  semicontinuity  of  J^(x)  as  a function  of  e and  x, 

L 

at  E = 0.  For  this  purpose  let  us  consider  initial  state  x 

tending  to  a limit  x*^  as  c 0 (x*"  c A)  . As  in  S2,  let 
c r. 

V , n be  the  optimal  control  process  and  corresiionding  solution 

t;  e t: 

of  (2.8)  with  n (0)  = X . Recall  that  v (t)  is  obtained  via 

f;  rv  t , 

(2.13)  from  the  feedbaclc  control  law  V , for  0 < t < 0 , and 

V ( t)  = 0 for  t > ^*  . 

Consider  any  sequence  e , tending  to  n as  n > «>,  and  let 
e r 

V = V , X = X , etc.  We  then  have 

n IV 


(4.1) 


n = ip  + , 

'n  ' n n ’n 


whore 


rt  rt 

' (t)  = X + V (s)ds,  ?_(t)  = 0[n„(s)ldw. 

n n Jq  n n J 0 


We  give  c”[0,'")  the  following  metric,  equivalent  to  uniform 


11 


convergence  on  each  finite  interval  (0,T]: 


= I 2 1+1  |].  • 

3 = 1 ' '3 


n T T 

For  r C C [0,"")  lot  r denote  the  sot  of  restrictions  <P  to 

[0,T]  of  functions  ^ r.  Then  is  totally  bounded  in 

C^[0,"’)  if  and  only  if  is  totally  hounded  in  C*^[0,T]  for 

each  T.  In  the  next  lemma  we  verify  the  Prokhorov  compactness 

condition  for  the  sequences  '^n'^n' 


Lemma  4.1.  Assume  that  the  sequence  J (x^)  is  bounded. 

Then  for  every  <S  > 0 there  exist  totally  bounded  sets  ^ 1 5 ' ^ 2 6 

such  that  P(<i>  c r,r)  > 1-  6,  P(C  c > 1-  6,  n = 1,2,... 

n lo  n 20 


Proof . Since  an  optimal  control  process 


(4.2) 


u 

'"(x„)  = E||J'L|n„(t),v^(t)l<lt  + en„(Vl). 


Since  41  > 0,  J is  bounded,  and  L(y,v)  > c|v-b(y) 

0 


for  some  c > 


• n 

0,  E I V 

Jo  n 


(t)  - b[nj^(t)J(  dt  is  bounded.  Since 


v^(t)  = 0 for  t > 0^,  this  implies 


E I |v  (t)  Tdt  < C , 
0 


for  some  C.^.  Let 


^1(S  ~ ‘J’  ^ 1'^  (s)  1^  < Cj2^(S  ^(t-s)  for  0 < s,  t < j). 


Then  is  totally  bounded.  By  Cauchy-Schwartz , 

|v  {t)|^dt  < C.2^<S  ^ implies  £'  1'?..  Hence 

P(+jj  / rj j)  < 2‘35,  j . 1,2, . , 

/ r^,)  < 6. 

The  existence  of  with  the  required  property  follows  from 

the  assumption  that  o is  a bounded  function;  see  [13,  Proposition  9). 

This  proves  Lemma  4.1. 

Let  Y = (1+0  ) and  note  tliat  0 < > <1.  Consider  the 

n n n - n 

2 n 

triple  '’n' ^n^  ' regarded  as  a C x [0,11 -valued  random 

variable.  By  Lemma  4.1  there  is  a subsequence,  denoted  again  by 

Yj,|)  , for  which  the  joint  probability  distribution  measures 

converge  weaklv.  By  Skorokhod's  theorem  [14]  there  exist 

('f ^ , Y^)  , defined  on  the  sample  space  = (0,1|  with  the 

same  distribution  as  ‘’n' ^n^  ' such  that  with  probability  1 

<}>  tend  in  d-metric  to  limits  <P  f ^ ‘^nd  > tends  to  a random 

n n n 

variable  Y as  n 

Let  n = ij’  + • Then  d(n  »'?)  ^ 0 witli  probability  1. 

n n n n n ^ 

The  exit  time  0 of  n (t)  from  A satisfies  > =0  (1+0  ) ^ 

n n n n n 

Thus  Y < 1 if  and  only  if  0 tends  to  a finite  limit  0 as 
n We  also  have  MO)  = x^,  Mt)  t A for  0 < t < 0,  and 

<i>  (0 ) c dA  if  9<'». 

For  n,  <[>  e C*^[0,"")  and  0 < y < define 
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h(4>,; 


,Y)  = 

•*  0 


L[n(t:)  ,<f  (t)  ]dt,  Y = 3^  , 


if  ^ e otherwise  let  h(n,<t',Y)  = +”.  Then  h is  lower 

n 

semicontinuous  on  C'  [0,“>)  x [0,1].  This  is  a well  known  result; 

it  can  be  proved  in  the  same  way  as  [7,  II,  Lemma  1.2,  p.  329]. 

The  random  variables  h(n  ,<?  ,Y  ) and  h(n  ,>p  ,y  ) have  the 

n n n n jn  n 

same  distribution.  Moreover,  with  probability  1 


(4.3) 


' n __  ^ i 

lim  inf  L ( n ,(})  )dt  > L (t]' , <}>)  dt. 

n-v-  Jo  " " "Jo 


Lemma  4.2.  Assume  (B^^)  . Then  for  any  x e A, 


lim  inf  J^(x)  > J(x^). 


Proof.  It  suffices  to  show  that 


lim  inf  J ^(x  ) > J(x  ) 
n - 

n-voo 

0 ^n 

for  any  sequences  tending  to  0,x  with  J (x^^) 

bounded.  Using  the  notation  above,  we  may  suppose,  moreover,  that 

tend  to  'F  in  d-metric  and  0^  > 0,  with  probability  1. 

In  (4.2)  we  replace  by  n and  recall  that 

nnn  nnn 

V = F . By  Fatou's  lemma 
n n 


(4.4)  lim  inf  J (x  ) > E lim  inf 
n*™  “ r,  >“ 


w 

1„ 


)dt  + ^1'^%)] 


Since  J "(x^)  is  bounded  and  <Ii  > 0,  (4.3)  implies 
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< « with  probability  1.  Since  (t)  e A for  0 < t < 0, 


0 


property  (B^^)  implies  0 < «>  with  probability  1.  Moreover, 
ri^(0^)  -*-0(0)  with  probability  1,  and  0(0)  e 3A.  From  (3.1), 
(4.4),  and  the  fact  that  'P  is  continuous  on  9A  we  then  have 


lim  inf  J ^(x  ) > E 
n-><»  ” 


,0)  . 


Since  0(0)  = x^,  we  have  ^(0,0)  > J(x^)  with  probability  1, 
and  hence  E _^(0,0)  > J(x^).  This  proves  Lemma  4.2. 


Note.  It  can  be  shown  that  J (x)  J(x)  as  e ->■  0,  although 
we  do  not  use  this  fact.  A proof  that  lim  sup  J^(x)  < J(x)  can 

e-*-0 

be  given  by  the  method  at  the  end  of  §7.  The  essential  idea  is 
that  given  0,  the  deterministic  control  v(t)  = (t)  can  be 

considered  as  a (suboptimal)  control  in  the  stochastic  problem 
in  §2.  A similar  comparison  technique  was  used  in  [3, §5]. 


5.  The  exit  problem  (preliminary  result) . Let  us  return  to  the 

exit  problem  for  solutions  C^(t)  to  (1.1).  In  the  present 

2 

section  we  consider  a bounded  open  set  A,  with  C boundary  3A , 
for  which  the  rather  strong  assumption  (B^)  in  §3  holds.  We  prove 
Theorem  5.1,  which  is  then  used  for  the  main  result  in  §6. 

For  N C 9A,  X e A let 


(5.1) 


q^(x,N)  = 


P^(C^(t^)  e N)  , 


I^(x,N)  = -e  log  q^(x,N) 


(5.2) 


I, 


with  U the  exit  time  from  A of  K*' it)  . Let 


(5.3) 


Ia(x,N)  = inf  L(>J)  (t)  ,<!)  (t)  ]dt, 

<t>  , 0 J 0 


where  the  infimum  is  taken  among  all  r and  0 such  that 

t (0)  = X,  4' (t)  r A for  0 < t < 0,  and  c N.  The  infimum  may 

also  be  taken  in  the  set  of  (4'/0)  for  which,  in  addition,  4>  (t)  c A 
for  0 < t < 0 (and  hence  0 is  the  exit  time  from  A for 
't’(t).)  We  also  have 


(5.4) 


I.  (x,N)  = I.  (x,N) 


For  M=  1,2,...,  we  introduce  a "penalty  function"  4'^^  i 0 such  that  4^ 

is  class  C^,  4'..(x)  ► as  M -+  uniformly  on  anv  comi’act  subset 
M ..1- 

of  - N and  =0  on  N.  The  following  lemma  is  easily 


proved,  using  lower  semicontinuity 


f 

itv  of  L(s',4’ 
0 


4')dt  with  respect 


to  'f'  in  the  d-metric  and  with  respect  to  0 . Write  J = J 

for  the  function  in  (3.2),  when  4*  = 4’... 

M 


Lemma  5.1.  Assume  (B^) . Then 

lim  inf  J^(x)  > I.  (x*^,tJ). 

0 

X ‘X 

Let  denote  the  interior  of  N,  relative  to  .3A 


Theorem  5.1.  Assume  (Bj^)  and  that  N = N . Then 

lim  I^(x,N)  = I^(x,N),  uniformly  for  x in  any  compact  sot  A C A, 
i:-»0 
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1 


Proof.  It  sufficos  to  show  that,  uniformly  for  x c A,  | 

(5.5)  lim  sup  I^(x,N)  < I. (x,N^)  ■ 

£>0  ” 

(5.6)  lira  inf  I^(x,N)  > I.(x,N). 

£->0 

Note  that  I,^(x,N^)  = I^(x,N)  by  (5.4),  since  N = N^.  Inequality  j 

(5.5)  is  an  easy  consequence  of  the  first  Ventcel-Freidlin  estiraate  I 

[7,  p.  332].  We  shall  indicate  another  proof  of  (5.5)  at  the  end  ] 

of  §7.  I 

To  prove  (5.6)  we  introduce  the  penalty  functions  above,  | 

and  write  when  (52).  Since  ~ ^ | 

C €M  I 

for  y e N,  we  have  by  (2.3)  and  (5.1),  q^(x,N)  < g'  (x)  for  each  M. 

GM  G 

By  taking  logarithras,  J (x)  < I^(x,N).  By  Leramas  4.2  and  5.1, 
inequality  (5.6)  holds  uniformly  for  x e A.  This  proves  Theorem  5.1. 

I 

6,  The  problem  of  exit  (continued) . Let  D be  a bounded  open 
2 

' set,  with  C boundary  3D.  We  illustrate  the  use  of  Theorem  5.1 

c 

by  deriving  an  asymptotic  result  about  the  exit  place  of  5 (t) 

(Theorem  6.1).  With  minor  variations,  the  construction  used  to 

prove  Theorem  6.1  is  the  same  as  in  [17,  p.  272] [7,  pp.  386-387]. 

_ 

For  X,  y E D let 

I 

f" 

L[<j>  (t)  ,<{>  (t)  ]dt 

° <p,0  Jo  I 

f 1 

, where  the  minimum  is  taken  among  c 0 > 0 such  that  i 

^ '(>(0)=x,'}>(0)=y,iJ>(t)ED  for  0^t<0.  By  (5. 3),  for  | 

L 
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X t D,  N C ?D 

I (x,N)  = inf  I_(x,y)  = min  I {x,y) . 
yi'N  ytN 

We  make  the  followinq  assumptions  (cf.  [7,  pp.  359-360]): 

(B^)  b(y)*v(Y)  < 0 for  every  y t'  ?d,  xvliere  '^(y)  is  the 

exterior  normal. 

(B^)  There  exists  a compact  set  K C D such  that: 

(i)  K contains  the  (^J-limit  set  of  the  solution 
K^(t;x)  of  (3.3)  for  each  x e D - K. 

(ii)  Ij^CXj^jX^)  = 0 for  every  ^ 

(iii)  Let  denote  the  M-neighborhood  of  K,  and 

A = D - K . Then  there  exists  c tending  to  0 as 
(I  -*■  0 such  that 

{x,y)  < Ip(x,y)  + c^^  for  all  x,  y e d - K2^. 

Assumption  (B^)  holds,  in  particular,  if  K consists  of  a 
single  point  x*  to  which  (3.3)  is  asymptotically  stable  from  n. 
By  (B^)(ii),  Ip(Xj^,y)  = I^(x2,y)  if  Xj^,  e K.  Let 

V = I (x,9D)  = min  I (x,y) , x e K, 
y e 9D 


a 


I = {y  £ 3D:  Ij^(x,y)  = V,  x c K). 
Theorem  6.1.  Let  (Aj^)  , (B3)  » (B^)  hold.  Then 


ii 

I' 

i 


dist(f;'^(  c^)  ,?;) 


0 in  probability  as  n 0. 


2 

Proof . Given  u ■ 0 let  S bo  open  witli  C boundary 

3S  and  K C S C K . Lot  A = D - S;  and  let  A = 

U 2\i 

(wo  take  n small  enouyh  that  C D.)  I.ot  N C 3D  be  closed 

with  ?;  C and  = n (N^  = interior  of  ,N  relative  to  an), 

T.ot  = 3n  - N.  There  exists  P 0 such  that,  for  x e K, 


Ij^(x,N)  = V,  Ip(x,N'^)  > V + 2P. 

By  (B^)  we  may  choose  n small  enough  that 

(6.1)  max  I.(z,N)  < V + P < mrn  I.(z,N^). 

zt.A  zeA 

Now  (B3)  f (B^)  imply  {B2)  . By  Lemma  3.1  (B^)  implies  (Bj^)  . By 

(6.1)  and  Theorem  5.1 

qNz,N'^) 

(6.2)  lim  -j; = 0 

‘■••O  q^(z,N) 

uniformly  for  z e A. 

r 

Given  X = (0)  vn  P,  we  define  random  times  ^ ,s  as 

n n 

follows 


II 

0 

j^St 

time 

t 

such  that  5^(t)  c 3 A 

s = 
n 

iSt 

time 

t 

■'  t , such  that  ^^(t) 
n-1 

c A (n  2 1) 

3 

II 

iSt 

time 

t 

•>  s such  that  ^(t)  r 
n 

3A  (n  > 1)  . 
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Consider  the  events 


e e, 


\ = f'n  = « '’d>  " ’'> 


t- , e. 


B.  = (C  = r(T^)  e nM. 


D 


By  the  strong  Flarkov  property 


e q^(ds^),N)] 


t >s. 


P^(B^)  = E^[x  , q^5'lsJ,N^)]  , 


'x“"  e -A 


with  Xq  the  indicator  function  of  an  event  G. 

d > 0 there  exists  g,  such  that 

d 

q^{z,N‘^)  < dq^(z,N) 


By  (6.2) 


for  all  z e A,  0 < E < e,.  Since  ^ (s,,)  e A,  we  have  P 

d ^ V ' X 

d P^(A^).  Moreover, 


I u B„)  = < .) 


= 1. 


Therefore,  for  0 < e < e^. 


P(^^(i!:)  G N^)  = I P„(B„)  < d. 


X'  V'  - 


Since  d is  arbitrary,  this  proves  Theorem  6.1. 


Note.  Theorem  6.1  is  slightly  less  general  than  [7,  II 


given 


(Bv)  < 


Theorem  7.2,  p.  361].  We  have  included  it  to  illustrate  the  use 


of  Tlieorein  5.1. 


Theorem  6.1  has  immodicite  aiiplications  to  solutions 
to  the  Dirichlet  problem  = 0 in  D with  u^'  = U on  9D 

(7,  II,  p.  3721.  For  instance,  if  J consists  of  a single  point 
y*,  then  u‘'(x)  U(y*)  as  t ->•  0,  for  all  x c d.  If  K consists 

of  a single  point  x*  t:  n,  then  results  equivalent  to  Theorem  5.1 
were  used  in  [17,  Lemma  1]  to  show  that 

lim  t;  log  F = v = min  I (x*,y). 

E ‘•0  X*  yt'9D 

7.  Finite  time  results.  Let  us  now  consider  the  nonautonomous 
form  of  (1.1),  on  a finite  time  interval  s < t < T: 

(7.1)  ds  = b [t , ^ (t)  Idt  + /e  o [ t , ^ ( t ) 1 dw 

with  initial  data  C^(s)  = x.  V7e  assume  that: 

(A2)  b(s,x),  o(s,x),  and  the  inverse  0 ^(s,x) 

r»  4-  1 

are  bounded,  Lipschitz  functions  on  r" 

We  fix  T and  consider  initial  data  (s,x)  in  the  cylinder 
Q = (-'",T)  X D.  Given  N C 9D  let 

fO 

(7.2)  I {s,x,N)  = inf  L [ t ,>|)  { t ) ( t ) ] dt , 

^ , 0 J s 

where  the  infimum  is  talcen  among  all  t e and  0 e [s,Tl 

such  that  >('(s)  = x,  >J>(t)  e D for  s<t<0,i{)(0)  eN.  The 
function  L is  as  in  (1.3).  Let 
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(7.3)  I^(s,x,N)  - -e  log  < T,  e N)  . 

Theorem  7.1.  Assume  (A2)  , and  tl^at  N = N*^.  Then  for  each 
( s , X ) e Q , 


lim  iNsjXjN)  = I (s,x,N)  . 

Z-yf)  ■ ^ 

Theorem  7.1  can  be  ['roved  by  the  same  method  as  for  Theorem  5.1. 
The  elliptic  equation  (2.1)  is  now  to  be  replaced  l)y  the  backward 

t; 

equation  0 = g^  + Ir  g , where 
(7.4)  g^(s,x)  = Egj^exp 

G C G £ 

where  = min(x^,T),  (s,x)  £ Q.  The  function  J = -e  log  g 

satisfies  the  corresponding  dynamic  programming  equation,  in  time 
dependent  form.  For  the  finite  time  problem,  assumption?  like 
(B^)  , or  (B^)  and  (B^),  are  unneessary.  However,  witliout  (B^) 
one  may  have  I^^(s,x,P)  = 0,  in  which  case  Theorem  7.1  is  un- 
interesting . 

For  the  special  case  N = 9D  we  have  (see  (7,  p.  347]): 

Corollary.  -lim  e ^’sx^'^D  ^ ~ Ij^(s,x;9D). 

A proof  of  the  Corollary  using  stochastic  control  ideas  was 
given  in  [ 5 ] . 

Instead  of  repeating  for  the  finite  time  case  details  of  the 
argument  in  §’s  2-5,  let  us  outline  a somev^rhat  different  proof 
of  Theorem  7.1.  A slight  refinement  of  the  method,  which  we 


r , c ^ , c , . T 

♦ |Tq,5  Vq)1 

e 
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shall  not  give,  shows  that  the  convergence  in  Theorem  7.1  is 
uniform  on  compact  subsets  of  Q U {(-'",T)  x N^). 

First  of  all,  lot  us  derive  the  finite  time  analogue  of 
Theorem  2.1  in  a different  way  using  the  Girsanov  formula  and  Ito 
stochastic  differential  rule.  Let  v(t)  be  a bounded,  non- 
anticipative  control  process  on  [s,T] , and  n the  solution  of 


(7.5) 


di)  = v(t}dt  + /e  o[t,n(t)]dw. 


Equation  (7.5)  is  obtained  from  (7.1)  by  replacing  the  drift  co- 
efficient h by  v.  Let 


h(t)  = a (t, n(t) ) (b (t,n (t) ) - v(t)] 


By  Girsanov' s formula  [7,  I,  Chap.  7] 


E exp  - 


= E exp 


rT 

- I j'l>(0,n(U))  t j |h(t)  l^dtj 

— s 


1 

+ — h(t) 'dw  , 
/e  J s J 


where  0 is  the  exit  time  of  (t,r)(t))  from  Q and  the 

_ e 

exit  time  of  (t,t  (t))  from  Q. 

Let  us  require  tliat  v(t)  = b(t,ii(t))  for  t > 0.  Then 


i|h(t)|^  = L (t , n (t) , V (t) ) , s < t < 0, 


and  h(t)  = 0 for  t > 0.  By  (7.4) 
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q (s,x) 


E Pxp(-X) , 


(7.6) 


X 


,n(0) ) 


L(t,ri(h)  ,v(t)  )dt 
^ s 


ny  Jonson's  inociualihy , K cxp(-X)  > oxi)|-E(X)l,  vvihh  equality  if 

and  only  if  X = constant  almost  surely.  However,  K(X)  = f {s,x,v) 

\tfhere  is  defined  as  in  (2.9).  Thus,  q ^ '] . To 

r f. 

qet  ec^uality,  we  define  v , q as  in  !*  2 usinq  tlie  feedjjack 
control  law  (cf.  (2.14)) 


v' (s,x)  = b(s,x)  - a(s,x) (J^)  '. 


i| 

,1-! 

|1! 


'I  i 

ir' 


.1 


Let  X^  bo  defined  »)v  (7.6)  when  n = n , v = v , 0=0^,  the  I 

exit  time  of  (t,n  (t))  from  Q.  We  apply  the  Ito  differential 

rule  to  (t,n  (t))  and  use  the  fact  thcat  J*  = on  9Q  to  ! 

'i 

conclude  that  X = ' J (s,x)  almost  surely.  Thus,  v (t)  is  ] 

an  optimal  control  process.  This  proves  the  finite  time  analogue  ■ 

of  Theorem  2.1.  '■ 

In  the  last  step  of  this  argument  we  have  used  the  fact  that  ; 

J*^(s,x)  is  of  class  C^'^(O)  O C^(0).  This  follows  from  the  fact  | 

that  g = oxp(-t  ) is  a positive  solution  of  the  linear 
paralnolic  ecpiation  0 - By  |12,  Chap.  IV,  §9), 

g^  r C^'^(Q)  n C^(())  provided  the  restrictions  of  'l'(s,x)  to 
(-",Tl  " iin  and  to  fT)  D are  C^. 

It  would  bo  interesting  to  avoid  results  from  the  theory  of 
parabolic  partial  differential  equations  altogetlncr  in  this 
argument.  Tliis  could  perhaps  be  done  using  methods  from  (2,S61 
or  |1,  Chap.  IV). 


I (r., 


I.ot  us,  for  brovity,  sot  I^j(s,x;N)  = l'  (s,x),  Tj^(s,x;N)  = 

In  order  to  prove  Theorem  7.1  it  suffices  to  sliow  that 

(7.7)  lim  sup  I^(s,x)  < T(s,x) 

L vO 

(7.8)  lim  inf  I^(s,x)  > T(s,x). 

‘ M) 

Let  us  first  outline  a proof  of  (7.8),  whicli  does  not  involve 
the  Prokhorov  comiiactness  criterion  and  the  Hkorokhod  theorem 
used  in  fi4.  by  introducinq  penalty  functions  as  in  !i5,  it  suffici's 
to  prove  the  followinq  Lemma  7.1,  which  replaces  Lemma  4.2.  Let 

>■=  ((-",T)  ao)  t)  ({Tl  X r")  . 


For  any  s < T and  x t r’^,  let 


(7:9)  vT(s,x)  = min  ( f L [t,><>  (t)  ,<?  (t)  Jdt  + f 0 (P  ) ] j , 

'I'  , l>  ' J s ' 


where  the  minimum  is  taken  among  all  '{’  <•  such  that  (s)  = x 

and  1)  such  that  (^*  jT  (^' ) ) ^ -^.  Let  j'  = -i  logg*. 


Lemma  7.1.  Assume  (^2^  and  that  i.^  Lipschitz.  Then 

lim  inf  j'^(s,x)  > J(s,x). 
t‘n 

Proof . For  t:  > 0 let  = b - ^(d^)  ’ )5C  the  optimal  feed- 

back control,  as  in  (2.14).  Given  (s,x)  t Q,  a brownian  motion  w, 
and  to  which  w is  adapted,  we  define  v*  (t),ii'  (t),'J’'  (t) 


as  in  § 4 bv 


£ . t ( 

n ( t ) = f ( t ) + /c  o ( r , tr  ( r ) 1 dw , 


9^(t)  = X + I v^(r)dr, 

^ s 

v'  (t)  = [t,  (t)  I , s < t < 0*^, 


with  ^ the  exit  time  from  Q of  (t,n^(t)).  F’or  t > 0^, 
v^(t)  = 0.  Lot 


(7.10)  = I L(t,  n^(t)  ,v^(t)  )dt  + <f(0^,r/(u‘^)  ) 


Let  us  show  that  for  some  constant  M 


(7.11) 


j(s,x)  < G^'  + m(  I ^ I I , 


where  ||  M is  t)ie  sup  norm  on  [s,Tl.  By  (1.3)  and  assumption 

2 

(A2),  L(t,y,v)  > Cj^|v|  - C2  with  > 0.  Hence  there  is  a 


constant  C,  such  that  v7  (s 

^ t 


c r*  r y 

,x)  < G if  |v  (t)  I dt  > C,  , 

Jr  ^ 


f L 2 

Suppose  that  |v  (t) | dt  < C. . An  elementary  dynamic  programming 
■ «?  ~ 


argument  shows  that 


(7.12) 


J(s,x)  < L(t,'))  ,<^  ) dt  + J (U  (u  ) ) 

J Q 


for  any  >1)  c with  I (s)  = x and  any  u t (s,T].  Moreover, 

there  exists  C2  such  that  x e r”,  y k 3d  imply 
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1 

I 

(7.13)  J(s,x)  < <^(s,y)  + C-|y-x|. 

i 

This  is  seen  by  taking  in  (7.9)  the  linear  function 

'P(t)  = X + v(t-s),  V = |y-x(~^(y-x) 

and  using  the  fact  that  is  Lipschitz.  If  ' s = T,  (7.13)  holds 

for  all  x,y,  since  J = 1'  when  s = T.  by  (A^)  and  (1.3)  there 
exists  such  that 

(7.14)  |L(t,y,v)  - L(t,z,v)|  < | y-z | ( 1+ | v | ^ ) 

for  all  y,z,  v e . By  (7.12)  with  u = 0 , 

,0  , 

J(s,x)  < L(t,<p‘  ,v")dt  + J(e^y‘'(u'") 

■'  s ' 

^ ,0^  _ I 

< + |L(t,<P^',v^)  - L(t,  n^,v‘')  |dt  + ’ 

^ 0 

+ J(u'^,<i’‘'(H'  ) ) - ^'(O'',ri^(0^)). 

We  use  (7.13)  with  (s,x)  replaced  by  (0  ^ (0 '^ ) ) and  y by 
n (0  ).  In  (7.14)  we  rejilace  y by  (T  (t)  and  z by  n (t)  . 

Then  (7.11)  holds  with  = C^(T-s+C^)  + . 

Since  v^(t)  is  optimal,  J^(s,x)  = Kg‘ . By  (7.11) 

J(s,x)  < J^(s,x)  + E|  I I . 

Since  o is  bounded,  E | | n -9  | | -♦  0 as  c v q.  This  proves 


Lemma  7.1. 
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From  Lemma  7.1  inequality  (7.8)  follows,  as  already  indicated. 

It  remains  to  consider  inequality  (7.7).  As  mentioned  in  §5, 
(7.7)  is  an  easy  consequence  of  the  first  Ventcol-Freidlin  estimate. 
Lot  us  outline  a somewhat  different  argument  to  get  (7.7).  Let 
fj*  = (_«, ^T)  X N®,  We  begin  with  the  following: 

there  exist  ^ 

for  all  ( s , X ) r.  A , 


Lemma  7.2.  Given  a compact  set  A C Q U fj* 

such  that  P ( < T,i,  (i,^)  c N^)  > exp(-Ct  ^) 

s X u D - 


0 < t 


Lemma  7.2  can  be  deduced  from  the  Girsanov  formula,  or  by  a 
direct  proof  (which  we  omit).  We  then  prove  (7.7)  as  follows. 

Consider  any  if  of  class  and  0 < T such  that 

4>  (s)  = X,  >)>  (t)  f:  D for  s < t < 0,  >)’  (0)  e N^,  <f  (t)  / D for  t > 0 . 
Let  n be  the  solution  of  (7.5)  with  n (s)  = x using  the 
deterministic  control  v(t)  = 9 (t) . Lot  be  open  relative  to 

Q,  with  (t,9(t))  t for  ^ f ^ A^  C Q UN*.  Now 

I^(s,x)  satisfies  in  Q the  dynamic  programming  equation 


0 


Moreover , 
(t, (t) ) 


“ ^s  ^ ^ ^ik^x  X "'i'l  fL(s,x,v)  + .v]. 

j,k=l  J j k V 

r C^'^(())  fA  C^(A^).  Let  (?''  be  the  exit  time  of 
from  A^.  As  in  (2.11) 


(7.1  ) 


I^(s 


L(t,ri^,9)dt  + l''(eSn‘'(3^)) 


)■ 


As  e + 0,  I I I I -*•  0 with  probability  1 and  0 in 

probability.  Moreover,  I^(t,y)  = 0 for  (t,y)  e N*;  and  by 


T 
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E D 

Lemma  7.2,  I (t,y)  <C  for  all  (t,y)  • A . Then 


) ) < cp[  (3‘',n'  (n' ) ) / n*1  , 


and  the  right  side  tends  to  0 as  i ‘ 0,  We  then  have  by  (7.15) 


(7.16) 


lim  sup  I ' ( s , x)  < 
i -0 


L ( t , q' ) dt . 


Since  the  infimum  of  the  right  side  among  such  ?,('  is  I(s,x), 
we  get  (7.7) . 

Note.  The  same  reasoning  can  be  used  to  prove  (5.5).  Given 
T > 0,  let  Q = (-"°,T)  xA  . Since  I,5^(>CfN)  < 1^(0, x,N),  (7.16) 
with  D replaced  by  A implies 


lim  sup  I. 
e->-0 


,0 

f(x,N)  < 1 

" J 0 


L(4'  ,l')dt 


for  any  ^ such  that  '(’(0)  = x,  <(>  (t)  t.  A for  0 < t < 3, 
ij)(6)  e (take  any  T ''  . The  infimum  of  the  right  side  is 

(x,N°) . 

A stronger  result  than  Theorem  7.1  (or  Theorem  5.1)  would  be 
an  asymptotic  expansion  in  powers  of  i of  the  form 


(7.17) 


. ^m„  . , m, 

. + e " + o ( E ) 

m 


Such  an  expansion  can  only  hold  in  regions  where  Iq  is  a smooth 
solution  of  the  Hamilton- Jacobi  equation,  constructed  from 
minimizing  curves  for  (7.2).  Such  regions  wore  called  in  [4] 
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